Demo: Exploring Data (Self-Paced)

This demonstration illustrates performing initial data exploration in three parts.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries,LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.


2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.


3. Click the Run button on the flow toolbar.

Part 1: Obtaining Summary Statistics for Continuous Variables

In this demonstration, you perform initial data exploration using a flow with the Summary Statistics node. You obtain summary statistics for continuous variables.

1. Open the previously saved flow Explore PVA.flw from the VST folder.


2. Under SAS Steps, expand the Statistics group, then double click on the Summary Statistics step.


3. Next, move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Summary Statistics node (the input port). Now Summary Statistics node is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.


4. Click on the Summary Statistics node, then select the Data tab. Next to Analysis variables, click the plus sign to add variables. Click the box next to Name to select all the variables. Click OK.


5. Click on the Option tab. The Mean, Standard deviation, Minimum value, Maximum value, and Number of observations are selected by default. Keep these options and additionally select Number of missing values.


6. Still under the options tab, expand Additional Statistics. Select Mode, Skewness, and Kurtosis. Expand Percentiles, then check off Lower quartile, Median, and Upper quartile.


7. Save the flow then right-click on the Summary Statistics node and choose Run node. Open the Results in a new browser tab.

• Notice that several observations have missing entries for the variables GiftAvgCard36 (Gift Amount Average Card 36 Months), DemAge (Age), and Donation_Amt (Target Gift Amount). The Donation_Amt variable has as many as 50% (4843 out of 9686) of its values missing. Also, skewness statistics indicate that most of the variables with the prefix Gift are positively skewed with skewness values ranging from approximately 1.1 to 14.


• Then you examine the DemMedIncome (Median Income Region) variable. Note that it has a minimum value of zero and a very large standard deviation (28707.49). This looks suspicious and warrants further scrutiny. A closer look at its mode (also zero), which is significantly different from its mean and median, reveals an important insight. That is, out of all the income values, zero is the most frequent. It appears that these are the cases for which an income is unknown and has been coded as zero (0). Clearly, having zero income is considerably different from having an unknown income.


• If you properly use the income variable in a predictive model, the solution to this situation is to recode these values as missing and then impute. This solution is presented in upcoming lessons.


• Notice the StatusCatStarAll (Status Category Star All Months) variable, which has a minimum value of zero and a maximum value of 1. In fact, it's a binary variable. By default, SAS treats every numeric variable as interval and every character variable as nominal. Going forward, we will ensure that we use this variable as a categorical variable rather than an interval variable.


• Among the other variables, let's examine the summary statistics of the interval-valued target variable Donation_Amt (Target Gift Amount). You see that the mean (101.64), median (102.6), and mode (92.10) are not very different from one another, with a skewness value (-0.086) being close to zero. These statistics give us the impression that the distribution of the Donation_Amt variable is fairly symmetric. However, it's always recommended to examine the distribution plots of variables to graphically assess the shape and spread of the data. Data satisfying normal distribution is beneficial for model building and is required in upcoming lessons on regression modeling. Next, you examine the distribution of continuous variables.

Part 2: Distribution Analysis of Continuous Variables

In this demonstration, you use a flow with the Distribution Analysis node to examine the distribution of continuous variables.

1. Return to the previous flow containing the Summary Statistics step (Explore PVA.flw). From the Steps menu, under SAS Steps, expand Statistics, then double click on Distribution Analysis.


2. Move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Distribution Analysis node (the input port). Now Distribution Analysis is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.


3. Click on the Distribution Analysis node, then select the Data tab. Next to Analysis variables, click the plus sign to add variables. Add all the variables except for StatusCatStarAll. A quick way of getting this done is by clicking the select all box next to Name and then deselecting StatusCatStarAll. Click OK to close the Column Selection window.


4. Click the Options tab. Verify that Histogram has been selected. Select the Add normal curve box under Exploring Data to include a normal curve for the histogram. To include specific summary statistics, select the Add inset statistics option and expand the Inset Statistics properties and select Mean, Standard deviation, Skewness, and Kurtosis in addition to the default of Number of observations. To assess normality using the normal probability plot, select the Normal probability plot box under Checking for Normality.


5. Right click on the Distribution Analysis node and choose Run node. Save the file Explore PVA.flw. When the run is complete, you can view the results on a browser tab. Click the More options on the right side of the screen (at the top right side of the open tab). Click Open in a browser tab and then click Results. View the results.

Note: If you see a warning message in the results window that says The maximum size limit has been exceeded, you might want to click the Open in new tab link to view the results. Alternatively, go to the SAS Studio Options menu, and select Preferences. Under SAS Programs, click Results. Under Display warning if results are larger than specific size, change the Maximum result size (MB) from 3 to 10.

Your results window displays histograms for all the analysis variables assigned in the Analysis variables property. In a practical scenario, it is worth spending time to examine the shape and spread of all the analysis variables in data. However, this is overwhelming in the class setup. So, let's examine the distribution of a few chosen variables. Consider the histogram for the Gift Count 36 Months variable.

The histogram and the normal curve superimposed on it indicate that the distribution is positively skewed, and the same can be ascertained from the statistics in the inset box. Scroll further down to examine the distribution of the DemMedIncome variable.

You clearly see a spike around the zero value, which accounts for approximately 25% of the income values. This resonates with the summary statistics, which indicate its mode value to be zero.

Now you consider examining the distribution of our interval-valued target variable: Donation_Amt (Target gift amount).

The histogram superimposed with the normal curve indicates that the variable is relatively normally distributed.

Next, to examine the normal probability plot for Donation_Amt, scroll down further until the bottom of page, and you see that the distribution of data points follows the normal reference line quite closely.

Thus, the data distribution looks fairly normal. This is an important insight about the interval valued target variable which could be useful in upcoming lessons on modeling.

Part 3: Generating Tables and Plots for Categorical Variables

In this demonstration, you use a flow with the One-Way Frequencies node to generate frequency tables and frequency plots from your data.

1. Return to the previous flow Explore PVA.flw. From the Steps menu, under SAS Steps, expand Statistics, then double click on One-Way Frequencies.


2. Next, move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the One-Way Frequencies node (the input port). Now One-Way Frequencies is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.


3. Click on the One-Way Frequencies node, then select the Data tab. Next to Analysis variables, click the plus sign to add variables. Add the variables StatusCat96NK, StatusCatStarAll, DemCluster, DemGender, DemHomeOwner, and Response variables.


4. No changes are needed in the Options tab.


5. Save the flow. Click right click on the One-Way Frequencies node and click Run node. Click on Submitted Code and Results, then click Results to view the report. Alternatively, click the More options on the right side of the screen (three vertical dots at the top right side of the open tab), then click Open in a browser tab, and finally click Results.

The flow displays the frequency table and bar charts for the requested variables. Note that the DemCluster variable has as many as 54 distinct levels. Do you see any challenges involving using a categorical variable that has as many as 54 levels as input for model building? Scroll further to examine the distribution of yet another target variable in our data set, Response. The frequency table and distribution plot clearly indicate that the data set has an equal proportion of responses and non-responses.

Do you see any relationship between proportion of non-responses (in the Response variable) and the number of missing values in Donation_Amt?